Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add merge_asof function to catalog #409

Merged
merged 2 commits into from
Aug 27, 2024
Merged

Add merge_asof function to catalog #409

merged 2 commits into from
Aug 27, 2024

Conversation

smcguire-cmu
Copy link
Contributor

@smcguire-cmu smcguire-cmu commented Aug 23, 2024 •
edited
Loading

Wraps the pandas merge_asof function as a catalog method, only available to use on the catalogs index columns (since other columns aren't sorted without shuffling) and without using the margin cache (since Margin Caches do not maintain sorted indices)

Closes #393

@codecov Codecov
Copy link

codecov bot commented Aug 23, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 92.59259% with 2 lines in your changes missing coverage. Please review.

Project coverage is 97.90%. Comparing base (57dca96) to head (55ede31).
Report is 3 commits behind head on main.

Files Patch % Lines
src/lsdb/catalog/catalog.py 81.81% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #409      +/-   ##
==========================================
- Coverage   98.00%   97.90%   -0.10%     
==========================================
  Files          44       44              
  Lines        1504     1531      +27     
==========================================
+ Hits         1474     1499      +25     
- Misses         30       32       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 23, 2024 •
edited
Loading

Before [57dca96] After [1738e42] Ratio Benchmark (Parameter)
failed failed n/a benchmarks.time_box_filter_on_partition
1.55±0.03s 1.56±0.02s 1.01 benchmarks.time_create_midsize_catalog
10.2±0.1s 10.3±0s 1.00 benchmarks.time_create_large_catalog
92.6±1ms 91.3±1ms 0.99 benchmarks.time_kdtree_crossmatch
26.9±0.3ms 26.4±0.7ms 0.98 benchmarks.time_polygon_search

Click here to view all benchmarks.

@hombit hombit linked an issue Aug 26, 2024 that may be closed by this pull request
Wrap Pandas' merge_asof #393
Closed
3 tasks
hombit
hombit reviewed Aug 26, 2024
View reviewed changes
Copy link
Contributor

@hombit hombit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it and it works fine!

However, I've got this warning when running the code. This message is misleading, because margins wouldn't work anyway with the current merge_asof implementation:

.../lsdb/dask/join_catalog_data.py:446: RuntimeWarning: Right catalog does not have a margin cache. Results may be incomplete and/or inaccurate.
  warnings.warn(

src/lsdb/catalog/catalog.py Outdated Show resolved Hide resolved
src/lsdb/dask/join_catalog_data.py Show resolved Hide resolved
@smcguire-cmu
Copy link
Contributor Author

I tested it and it works fine!

However, I've got this warning when running the code. This message is misleading, because margins wouldn't work anyway with the current merge_asof implementation:

.../lsdb/dask/join_catalog_data.py:446: RuntimeWarning: Right catalog does not have a margin cache. Results may be incomplete and/or inaccurate.
  warnings.warn(

That's me misusing copy and paste, fixed now!

hombit
hombit approved these changes Aug 27, 2024
View reviewed changes
Copy link
Contributor

@hombit hombit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@smcguire-cmu smcguire-cmu merged commit 67bcf5a into main Aug 27, 2024
10 of 12 checks passed
@smcguire-cmu smcguire-cmu deleted the sean/merge-asof branch August 27, 2024 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hombit hombit hombit approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Wrap Pandas' merge_asof
2 participants
@smcguire-cmu @hombit